TNO.iTRIS-lOOlCON 



PATENT 



IN THE UNITED STATES PATENT AND TRADEMARK OFFICE 
In re application of: Meisner et aL 

Serial No.: 10/664,565 Group Art Unit: 2625 

Filed: September 18, 2003 Examiner: Azarian, Seyed H. 

For: AUGMENTED REALITY TECHNOLOGY 

Certificate of Mailing 
I, Rebekah L. Mitchell, hereby certify that this correspondence is being 
deposited with the U.S. Postal Service in an envelope addressed to the 
Assistant Commissioner for Patents, Washington, DC 20231 on the 
date shown below. 

On: January 17.2006 

By:. 



Rebekah L. Mitchell (Person Mailing Correspondence) 



Honorable Commissioner of 
Patents and Trademarks 
Washington, D.C. 20231 



Sir: 



RULE 131 DECLARATION 

I, Walter Donnelly, hereby declare: 

1 . I am the Chief Executive Officer of the TriSen Augmented Reality Company, which is the 
assignee of the above-captioned utility patent application. I am also one of the three 
inventors named in the application. 

2. I am executing this Declaration on behalf of the TriSen Augmented Reality Company as it 
is not presently possible to locate all three coinventors. 

3. The document that is attached hereto as Exhibit A is known to the TriSen Augmented 
Reality Company and me to have been drafted by my coinventor, Jeffrey Meisner, prior to 
or on August 6, 1995 and is titled: "Image Transformations in a Videometric Tracking 
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System." This was a confidential internal document belonging to TriSen Systems and was 
not disseminated to the public. It describes the original design specification and 
architecture for the TriSen augmented reality system, 

4. The document that is attached hereto as Exhibit B is known to the TriSen Augmented 
Reality Company and me to have been drafted by Mr. Meisner prior to or on August 1, 
1995 and is titled: "Analysis of Error in the Videometric Tracking System" This also was a 
confidential internal document of TriSen Systems and was not disseminated to the public. 
This is the original error analysis document for the TriSen Videometric tracker. 

5. The document that is attached hereto as Exhibit C is a progress report that is known to the 
TriSen Augmented Reality Company and me to have been prepared by TriSen and used in 
a conference that was held on November 2, 1995 to report progress, in confidence, to the 
current prime contractor, Honeywell. 

6. The document that is attached hereto as Exhibit D is a paper copy of an e-mail message 
dated March 26, 1996 that is known to the TriSen Augmented Reality Company and me 
to have been sent fi'om TriSen to Honeywell reporting interim status m converting the 
code for the working TriSen tracker fi-om the Forth programming language to C. 

7. The document that is attached hereto as Exhibit E is an interface specification that is 
known to the TriSen Augmented Reality Company and me to have been prepared by the 
inventors prior to or on July 9, 1996 for the second VFE design, which used a parallel 
port to communicate with the wearable computer. 

8. The document that is attached hereto as Exhibit F is known to the TriSen Augmented 
Reality Company to have been drafted on or before October 30, 1995 and is the inventors' 
original hand drawn design for the original VFE, including the equations used in the 
programmable gate array logic device, showing some development already completed. 

9. All of the attached Exhibits were drafted within, and the events referred to occurred 
within, the United States of America. 
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10. The attached documentation, and Exhibits A, B, C and D in particular, demonstrates that 
the inventors prior to August 2, 1996 conceived a method for tracking the position and 
orientation of an object as described in independent claim 1, specifically that included 
steps of (a) scanning across an object to detect fiducials, wherein a video run is formed by 
a scan; (b) clumping video runs to detect a pattern of fiducials; (c) acquuing estimated 
values for a set of tracking parameters by comparing a detected pattern of fiducials to a 
reference pattern of fiducials; and (d) iterating the estimated values for the set of tracking 
parameters until the detected pattern of fiducials match the reference pattern of fiducials to 
within a desired convergence. 

11. Exhibit C, Page 8 describes a process where a video run is formed by a scan, and wherein 
video runs are clumped to detect a pattern of fiducials. The "tracking correspondence and 
estimation" on page 8 of this document describes a process of acquiring estimated values 
for a set of tracking parameters by comparing a detected pattern of fiducials to a reference 
pattern of fiducials. 

12. Exhibit F describes in detail a process of acquiring estunated values and iterating the 
estimated values for the set of tracking parameters until the detected pattern of fiducials 
match the reference pattern of fiducials to within a desired convergence. 

13. The attached documentation, and Exhibits A, B, C, D and F in particular, demonstrates 
that the inventors prior to August 2, 1996 conceived a method for augmenting reality as 
described in independent claim 1 1, including steps of (a) tracking the position and 
orientation of a pattern of fiducials on an object with a self-contained, mobile system; (b) 
processing virtual information stored in a computer memory of said system according to 
the position and orientation of the object; and (c) presenting the virtual information with 
real information to a user in near real time with the system. 
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14. The user interface for receiving input and presenting augmented information to a user, and 
which includes a head mounted (mobile) display on which the computer presents the 
augmented information is described on page 3 of Exhibit C. 

15. Exhibit C, Page 8 describes a process where a video run is formed by a scan, and wherein 
video runs are clumped to detect a pattern of fiducials. The "tracking correspondence and 
estimation" on page 8 of this document describes a process of acquiring estimated values 
for a set of tracking parameters by comparing a detected pattern of fiducials to a reference 
pattern of fiducials. 

16. Exhibit F describes in detail a process of acquiring estimated values and iterating the 
estimated values for the set of tracking parameters until the detected pattern of fiducials 
match the reference pattern of fiducials to within a desired convergence. 

17. The attached documentation, and Exhibits A, B and C in particular, demonstrates that the 
inventors prior to August 2, 1996 conceived a method for augmenting reality as described 
in independent claim 26, comprising steps of (a) using a sensor to provide at least one 
signal that is indicative of a pattern of fiducials on an object; (b) processing the signal to 
locate the fiducials; (c) determining a relative position and orientation of the sensor with 
respect to the object by comparing the locations of said fiducials to a known reference 
pattern; (d) providing virtual information to a user in substantial registration with real 
information based on the relative position and orientation determined in step (c), and 
wherein the method is performed so as to provide the virtual information to said user in 
near real-time. 

18. More specifically, the optical sensor that is referenced in several of the claims is described 
in several of the Exhibits as a video camera. For example. Exhibit A, p. 1 mentions a 
video camera in several places. Exhibit C lists the specific hardware that was used in the 
invention as it was contemplated on November 2, 1995. 
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19. Exhibit C, Page 8 describes a process where a video run is formed by a scan, and wherein 
video runs are clumped to detect a pattern of fiduciais. The "tracking correspondence and 
estimation" on page 8 of this document describes a process of acquiring estimated values 
for a set of tracking parameters by comparing a detected pattern of fiduciais to a reference 
pattern of fiduciais. 

20. Exhibit A, p. 1 specifically describes an image containing a number of fiducial markings, 
and states that the fiducial markings are on an object plane and that the positions of the 
fiducial markings are known with certainty. 

21. The calculation of angular parameters including phi, theta and psi, wherein said position of 
said at least one sensor with respect to said pattern of fiduciais is defined by Imear 
parameters including X-bar, Y-bar, and distance, said hard fiducial pattern being used to 
identify said phi, theta, psi and distance parameters and said soft fiducial pattern bemg 
used to identify said x-bar and y-bar parameters is described in Exhibits A and B. 

22. The step of processing virtual information stored in a computer memory according to the 
position and orientation of the object is described in Exhibit C, p. 9. P. 4 of this document 
describes the presentation of virtual information with real information to a user in near 
real-time. 

23. A prototype that would be read upon by the independent claims presently pending in the 
above captioned application was manufactured, reduced to practice and demonstrated to 
Honeywell in July 1996. It was later delivered by TriSen to the Boeing Corp. in mid- 

^ 1996. The undersigned is still attempting to recover documentation relating to this 
prototype, 

24. TriSen continued to develop and refine the technology in the period of time between the 
conception described above and the filing of the provisional patent application upon which 
the above captioned patent application claims priority. The pace of such development was 
complicated somewhat by the death of the president of TriSen and the departure of Mr. 
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Meisner from the company. However, the company remained diligent in reducing the 
invention to practice throughout this period, 
25. 1 hereby declare that all statements made herein of my own knowledge are true and that all 
statements made on information and belief are believed to be true, and further that the 
statements were made with the knowledge that willful false statements and the like made 
are punishable by fine or imprisonment, or both, under Section 1001 of Title 18 of the 
United States Code and that such Avillful false statements may jeopardize the validity of the 
application or any patent issued thereon. 



Respectfully submitted, 




By Walter Donnelly 
Chief Executive Officer 



Date: 




EXHIBIT A 




IjDage Trauasf ormat^ -^ns in a Videometric Tiackijig Syateai 

J,. ?ic;isner, TriSen Systems 

August 6, 1995 . 

In a videometric tracking systero, an object plane defined in 3- 
space as z:=0, is imaged onto the image plane of a video camera.. The 
image plane coordinates axe denoted (u.v).. The caffiera is positioned at 
Xo, y-o, 2o. and is first rotated horizontally in the x-z plane (yaw) by 
an angle 0, then rotated downwards (in 
the -y direction) by an angle 4> 
(pitch) , and then rotated about its own 
axis by an angle The resultant 

iraage is therefore a function of the 
markings ori the object plane and the 
six tracking parameters described 
above. Additionally, the image 
transformation is dependent on the 
scale (or equivalently the camera's 
field of view, or lens focal, length) i 
which does not change. 
We would like to explore 
the general character of 
mappings from one two- 
dimensional, surface to 
another, and consider 
solutions to the 
tracking problem given 
an image so obtained, 
containing a number of 
fiducial markliigs- on the 
object plane whose 
positions on the object 
plane are known with 
certainty.. 



Figure 1. Gi id pattern as seen 
straight on (all rotation angles are 
zero). 




We 



will only 



Figure 2.. Grid pattern of Figure 1 subject to general 10 
D..F. transform. Hote that lines in ohject plane are 
curved in image. 



consider bi-linear transforms from the (x.y) object plane to the (u,v) 
image plane.. The most general bi- linear transform is described as 
follows. 



/ 



u 



(1) 



d"x + e"y + 1 



// 



V 



(2) 



This very general transform is seen to have 10 degrees of freedom.. "In 
Figure 1 a grid pattern is shown as it has been simulated on an object 
plane (seen straight on. thus without distortion).. An arbitrary 
transform of the form of <1) , (2) has been performed.. The result is 
shown in Figure 2.. Note that straight lines on the object plane have 
become curved, a phenomenon that will not occur using distortion-free 
optics. 

As opposed to the 10 D. F. transform considered in (1) , (2), a 
plane projection transform, which describes the image formed by a 
telescopic lens at a large distance, is described by only six 
parameters. Setting d '=e '=d"=e" = 0 . we obtain simply; 



This therefore approximates a narrow f ieid-of-view lens, in which the 
ratio of areas between the object plane and the image plane is constant 



(3) 



(4) 
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throughout the image (as can be 
easily verified by computing .he 
Jaccbian of the trans f oriu) . The 
g.id pattern in Figure 1 has been 
J -i-v i» AS* downward 
transformed with a -o 

view in Figure 3 and -^^^ ^ f 
horizontal rotation followed by a 
45* downward rotation in Figuxe 

rAtio areas between , ^ 

plane is constant, thez. 

ambiguity ' 



iJerprataUon of the transformed 

iroage In Figure 3. ^-.f^^J; 
l»PO.slW.e to tell 
*=..46-, »e are only le« to 
„.^erve the co-pression of the 
verti<=al axl. according to the 
cosine of 




nls fUi. la not present m ^ 

3^ l„age obtained using a finite L_-_jJ^ ^„,.„Uon vle«d «th 

proiectlon. obtained »ith a =a»era ^45.. ♦=45 . 

having a substantial field of ^ ^ .a. 

For instance in F.g«e 5 the ^^^^ ^ ^^^^^ ^^^^^ 

again »ith a doonward y^'" the proiection of areas . 

Horizontal f leld-of -.lew is 90 tie a-biguity in 

1 ;«d ri::-::;. - - .o»n object, 

the sign of ♦ obtained by .^distinguishable from the 

The finite proieotion transforation as m^^^^^^ ,„„tralnts on 
arbitrary bilinear t--*-«*;°" °' ^''^.tl^ular , the deno-^nators of 
the values of the ten parameters. In Pa ^ ^^^^^^^^^^ ^ 
the "P-slons for u and ^^^^^ be transformed .0 

which insux-es that lines 
lines in the image plane. 



u = 



dx-+ ey+ 1 



+ ey + 1 



(5) 



(6) 



Although (5), (8) contain 8 parameters, there are only six actual 
degrees of freedom in the transformation, corresponding to the three 
positional parameters and the three rotational parameters we are 
tracking. In terms of the tracking parameters, the 8 coefficients in 
(5), (6) are given by: 



Sr(cosT|r cos0 + sinijr sin4> sin6) 
b' - g(simir cos<|>) 

= sr(xoj^osi|r cos6 - sinijr sin4> sin6) 

jyi^ +Zq {sinf sin^ cosO - cosi|r sin6) ) 
/r Q^u - g( -sinijr cos9 + cosijr sin<|> sinO) 
jb'' - ^cosi|f cos<t) ' 

c" = (sinijr cos6 ~ cosT|r sin(|> sin0) 
- yocosi|r cos4> 

+ {sim|r sin0 + cosi|r sin<l> cos0) ) 
c? » 5rcos<|>sin8 

where: 

, 1 . 

^ -Xq cos<fr sind + yo sin<> + cos4> cos0 



(7) 




Although observation of the transformation (as viewed^ for 
instance, in Figure 6) unambiguously determines the six tracking 
parameters, there is still an ambiguity when only observing the position 
of only thi-ee fiducial points. Even though three points on th^_. image 
^ plane, having six degrees of freedom, are sufficient to solve for. the 
six tracking parameters, there are still two solutions that satisfy the 
transformed positions of only three points. Thus a tracker observing 
only three fiducial points is capable of "latching on" to the wrong 
solution and diverging from the correct estimation. This will not occur 
when four or more fiducial are within the camera's f ield-of-view. 

Since four fiducials are necessary to unambiguously determine the 
tracking parameters, it may be possible to perform an estimation 
algorithm not on the above model having 6 degrees of freedom, but rather 
on the 8 degree of freedom aphysical model. The 8 D- F model is also 
described by <5), (6). however the constraints of (7) are no longer 
imposed.. While this model does not correspond to a physical paradigm, 
it is simpler than the 10 D. F model of (1), <2) , and guarantees that 
lines in the object plane are transformed to lines in the image plane- 
An estimation of* the 8 parameters of (5) , (6) . can be followed by 
collapsing the estimation into a 6 D. F.. model consistent with (7), thus 
retrieving the tracking parameters.. If four fiducial points are being 
observed and the eight parameters of (5), (6) are then solved, it is 
expected that the derivative solution of the six tracking parameters 
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would match those obtained by an estimation algorithm based on only the 
six tracking parameters.. However the initial estimation procedure may 
be simpler since trigonometric functions are no longer involved in the 
8^. F. model.. 



Par^jaeter Sstimat ion using the Gradient Descent Method 

Let us suppose that we have observed at least four fiducial points 
in the image plane at coordinates Ui,v,; u^.y,; etc. Suppose that the 
positions of the fiducial marks on the object plane are known to be 
Xi,yil ^zitV^t etc. Then, for a given 8 D. F.. transformation, we would have 
expected to observed these fiducials at positions: 

i Li (8) 



s — i ii 

^ ^ dxj^ + ey^ + 1 



(9) 



Then we would find an estimate of the eight parameters which minimized 

the net cost function Q. defined as the sum of the squares of the 
residuals. 

O = E «?i>Ui)'+ (10) 

X 



Then the gradient of Q is found as the 8 element vector formed of the 
partial derivatives of Q with respect to each paraiMter- 
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-M = P2 ^ — — i^i-u.) 

^0 - 



(11) 



^ = y 2 — — - v.) 

=: __2 {f^. - V.) 

ad 



(12) 



(13) 



<14) 



(15) 



(16) 



? L [ (u - Q ) Q + (v - ^j) (17) 



18) 



Multiplying the negative of the gradient of the cost function by a gain 
parameter, and adding that quantity to the parameter vector, will result 
in a iterative procedure which will solve the 8 D.F- estimation problem 
in a least- squares sense. 
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EXHIBIT B 




Analysis of Error in the Videometric Tracking System 
J . Meisner , TriSen Systems 
August 1. 1995 



A videometric head tracking system has been proposed in 
which a video camera mounted on a helmet will observe fiducial 
marks on a planar board in order to estimate head position for 
the purpose of driving a head mounted display (HMD) in order to 
superimpose computer- generated symbols to appear on the same 
plane as the fiducial marks The pertinent error measure is not 
the error in the measurement of head position or orientation, 
but rather the net error in the position of the computer-- 
generated symbols as they are superimposed on the plane at which 
the fiducials reside (hereafter: object plane).. 

The ensuing analysis will consider four different coordinate 
systems which will now be defined. There is a fixed three- 
dimensional coordinate system denoted x, y, z, relative to which 
fiducials (and intended symbol placement) exist in an absolute 
sense. The position of the (nominally) planar surface is 
defined by z=:0.. The y coordinate may be thought of as 
vertical, and the x coordinate as horizontal, along the surface 

A translated and rotated three dimensional coordinate 
system, denoted x\ y\ z\ is defined by the position of the 
pupil of the camera mounted on the helmet The origin of the 
primed coordinate system in the fixed coordinate system is 
located at. x., y«, 2,.. Following translation to that point, a 
rotation given by the orthogonal matrix B yields the primed 
coordinates. The two three-dimensional coordinate systems are. 
thus related by: 
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Then there are two two-dimensional coordinate systems to 
consider. One is simply the coordinate system of the object 
plane (on which the fiducials reside) which uses the coordinates 
X (horizontal) and y (vertical). These are simply the same x 
and y used in the fixed 3-D coordinate system, at which the 
object plane was stipulated to exist at z=0. 

Finally there is a two-dimensional coordinate system 
corresponding to the image received by the camera. For the 
purposes of the following analysis, we will consider the pupil 
of the eye to be located at the position of the pupil of the 
video camera, thus this coordinate system is also directly 
mapped to the visual field of the human. The coordinates of 
this system, are labeled u. v. where u is (for upright head 
positions) approximately horizonai (parallel to x'), and v is 
approximately vertical (parallel to y'). 

We define the scale of this coordinate system such that the 
tangent (not sine) of a deflection angle is given by *'(u'+v'). 
In particular , using a lens of focal length f . on the physical 
image plane x",y". where x" and y" are anti- parallel to x'.y'. 
(due to image inversion), u and v are defined by u=x'Vf and 
v=y"'/f. Then, if an object at a point in space (in the rotated 
3-D coordinatja system) exists at a position x'.y'.z', it will 
be imaged at u.v, such that: 



= LU 

y' = Lv 
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L, above, is a parameter expressing the distance between the 
camera pupil and a plane normal to the axis of the camera which 
contains the point x\v\z' . Siraply given an image in the u,v 
coordinate systera, L is .a priori unknown.. However given a 
transformation between the fixed and the primed coordinate 
systems, and knowing that an imaged point is located on the 
planar board at 2S=0, L can be solved for, thus obtaining the 
absolute location of that imaged point using 2 and 1„ 

A few physical details related to the image plane will be 
introduced at this point The coordinates u and v will be 
considered to run from -» to », thus covering a 18 0* field of 
view. Within this space, a physical video camera will have a 
finite field of view in which u and v are bounded by -l^ov and 
+ lpov- Of course most TV cameras have a 4:3 aspect ratio, 
however that detail will be ignored (or perhaps averaged) in the 
specification of l^v Even though the field of view of the 
camera itself is limited, we will be considering the position 
of a point of interest at further locations on the u,v plane. 

Now, a video camera with a resolution of R pixels across 
the above specified field of view, will have a pixel size in 
the image plane of Au= Av=21pov/R If we consider the measurement 
error e, in the u.v coordinate system, to be 1 pixel r. m. s- in 
each u and v, then e=2y21pov/R On the basis of this 

relationship, we can define an effective resolution R,^, for a 
camera which, has a net r ra s. error e in the determination . of 
the positicoii^^of an imaged fiducial point. For a camera with no 
appreciable* lene field distortion, fiducial image sizes of a few 
pixels, and '' effective centroiding, we would expect R.^^ to be 
close to (or somewhat less than) the actual hardware pixel 
resolution of the camera. 
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Analysis of Positioning Error in the Image Plane 



Given a field containing several fiducials in which we 
assume nominal measurement error, and a point of interest P 
in that field whose location is specified with respect to those 
fiducials, let us determine the resultant error in the 
determination (or placement) of the point of interest P Note 
that this discussion only pertains to the positional errors as 
seen in the image plane. In other words, we are only- 
considering the positional errors as they would appear with 
respect to the television image returned by the video camera, 
with no regards to the consequent positional error in the object 
plane (or in 3-space).. The propagation of these errors measured 
in the image plane to positioning errors in the object plane 
will be considered at a later time. 



In order to 
determine the magnitude 
of positioning errors 
6ue to imprecise 
measurement of the 
fiducial positions, we 
will assume that there 
are N fiducials (where 
N^3) in the field of 
view of the camera, and 
that the area- defined 
by this array of 
fiducials centered 
abo ut u= 8 , .* 9 , and of 
approximate extent 1 in 
both u and v. Since 



p\ 

























Figure 1. Triangulation (in image plane) of Point 

of Interest P. using four fiducials, possible due 

the magnitude of errors ^^<^^ ^hat lines in object plane transform to 

^ ^, lines in image plane., 
is not sensitive to the 

specific positioning of fiducials relative to P, we can just as 
well consider the special case in which there are 4 fiducials 



in a pattern in which the line defined by one pair of 
fiducials, Fi and F^. and the line defined by the other two 
fiducials, F3 and F^. conveniently intersect at the point of 
interest P. In other words. P is triangulated by the lines 
formed by each pair of fiducials as shown in Figure 1. This 
is an appropriate model to apply to the eventual errors that 
would be obtained in the object plane, since a point which is 
triangulated by two such lines in the image plane, would also 
be triangulated in the object plane by two such lines also 
defined by the two pairs of fiducials as they exist in the 
object plane (since lines in the object plane are imaged as 
lines) 

As can be seen, the geometzy specified in Figure 1 
corresponds to the point of interest lying well outside the 
camera f ield-of -view: the case of positional extrapolation We 
will limit our error analysis to such a geometry since 
extrapolation corresponds to the ' woist case" error result. 

So let us assume that each of these four fiducial positions 
are determined with a r.m. s. error in the image plane of e 
Assuming isotropic errors, that would imply errors in u and v, 
each of magnitude 6*^2/2 Then it can be seen that the error 
A(J> in the orientation of either line triangulating P in Figure 1 
would be given by: 



Additionally, the error Ap in the positioning of that line, in 
the direction normal to the line itself, would be: 



5 



Then it can be seen that the net exroi in the triangulation of 
P using lines with errors given hy 3 and 4 would be, in the 
direction normal to the line connecting P to the origin: 




where |P| denotes the distance of P from the origin in the 
image plane. The error in the triangulation of P parallel to 
the direction of the line connecting P to the origin, would be: 



In the extrapolation case where |Pt>l, the second error 6 is 



contribution of either error towards the resultant positional 
error in the object plane, will depend on the coordinate 
transformation, which is dependent on the actual 3-dimensional 
geometry, not considered in the above examination.. 

Now, 5 and 6 are based on the triangulation of . P using 
four fiducial positions. For an increased number of fiducials, 
a positional determination in which the independent errors due 
to each were averaged would decrease the net error by the square 
root of N, the number of fiducials employed. Thus, for N 
fiducials, we would rewrite 5 and 6 as: 




clearly of greater significance. 



However the specific 
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The Coordinate Transformation 

Let us no;^ look at the coordinate transformation between 
the two-dimensional iraage plane coordinate system, and the. two- 
dimensional object plane. This transformation is a result of. 
and unique to, the specific position and orientation of the 
video camera in space.. That includes three degrees of freedom 
in position and three degrees of freedom in describing the 
. camera's orientation- However we would like to identify the 
significant degrees of freedom which generate a change in the 
object plane error due to errors in the image plane.. In that 
vein, we will find only two significant degrees of freedom- 
Consider first the position of the camera's pupil, given by 
a position x^.y^^z^ in space. Of these 3-space coordinates, x 
and y are parallel to the object plane, and a simultaneous shift 
^in the camera position and the fiducial positions (or the 
presence of similar fiducials in the new position) will leave 
the problem of error propagation unchanged. However the 
coordinate z« describing the distance between the camera and the 
physical position of the object plane, clearly is of 
consequence. 

The orientation of the camera in space is given . by three 
angles. Two of these angles describe the direction of the axis 
of the camera. The third angle describes a rotation of the 

7. 



camera about that axis.. That rotation ig equivalent to a 
rotation of all points in the image plane (as described in the 
previous section) and has no further significance. The other 
two angles do alter the coordinate transformation in a 
significant way. However the qualitative changes in that 
transformation are clearly dependent on the angle made between 
the camera s axis and a norraal to the object plane, but not on 
the orientation of the plane which includes the camera's axis 
and the noimal to the object plane So there is only one 
significant degree of freedom in camera orientation, relative to 
analysis of the propagation of positional error. 

Therefore, we shall, without loss of generality, model the 
coordinate transformation as being a result of a camera whose 
pupil is located at x=0, y=:0. and z=Zg, and whose orientation is 
altered only by a pointing of the axis of the camera dotmward 
(in the -y direction) by the angle ^. For this special case, 
the reverse transformation, from the image plane to the object 
plane, can be simply written as: 



U Zr, 



vsxn^ + cos<(> 
vZq cos4> - ^oSin<(> 



Then, the sensitivity of errors in the object plane to errors 
in the image plane, is given by the (partial) derivatives of the 
coordinate transformation 9 with respect to the image plane 
coordinates.. 

Let us consider the point of interest P, to be found on 
the image plane at a point u,v. Let us call D the distance in 
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3-space between the camera pupil and the actual point of 
interest P on the 2=0 plane. In terms of u,v and the 
transformation parameters, D can be expressed as: 



D = 



vsincf) + cos$ 
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However we can just view D as an essential parameter expressing 
the distance between the tracking hardware (camera) and the 
work, ' 

Then the derivatives of the reverse coordinate 
transformation, evaluated around '.i.v. the point of interest, can 
oe shuwn to be given Dy: 



h2£ 

'6u 



D 



11 



-u sin<J) 
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6y 
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5v 



(1 + + v^) 
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ru:: ^:ixecl derivatives (considering that x is nominally 
parallel to u, and / is norainally parallel to v) given by 12 
and 13, are dominated by the other two derivatives, and will 
henceforth be ignored.. 



Calculation of Net Error in the Object Plane 

Using the error in the image plane calculated earlier in 
7 and 8, and using the sensitivity of errors in the object 
plane to errors in the image plane given by 11 and 14, we can 
find the net errors as measured in the object plane.. Thus we 
will find the positional error in the placement of computer- 
generated symbols as they appear on the plane at which their 
registration is defined.. 

The worst case net error occurs when the point of interest 
P is offset from the u,v origin in the same direction as the 
tilt defined by (fr. We then obtain the net error in the object 
plane by multiplying the error in the image plane given by 8, 
by the derivative of object plane position with respect to image 
plane position* given hy 14 We thus obtain for the error in 
the object plane AL, the following: 

AL - Ad ^ - eJ^-^d + ^) — — i5 
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For simplicity, let us examine the special (but not atypical) 
case when axis of the tracking camera is normal to the physical 
object plane: <(>=0, Let us call L the physical distance on the 
object - plane between the point that the axis of the camera 
intersects the object plane and the physical point of interest 
P.. Then 15 can be simplified by applying 10 and the definition 
of L„ 
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So for the case of <^0, 16. 20 says that the error on the 
object plane in locating a point of interest P, located a 
distance L from the point at which the axis of the camera 
intersects the object plane » is given as the product of three 
factors times L.. The first factor, e/1, in 16, 20. can be 
viewed as the reciprocal of the number of resolvable points in 
between fiducial markings.. The second factor is on the order 
of unity, decreasing slightly for a larger number of fiducials 
in the camera's field of view. The third factor is one plus 
the extrapolation coefficient, defined as the ratio of L to the 
distance between physical fiducials on the object plane.. 

As a hypothetical example, consider a video camera with a 
wide angle lens having a 90* field of view Suppose that the 
errors in the image plane are about 1 pixel, for a resolution 
of 500 lines. If the camera is located at 20 cm from the 
object plane, its field of view, at 4t:0. will subtend a square 
on the object plane of 40 cm size (with each pixel subtending 
a square of .8 ram) Suppose that a field of 4 fiducial marks 
covers 3/4 of this size: lz<,-30 cm.. Then if the point of 
interest is located at a distance of 50 cm below the camera, we 
would calculate the net error to. be: 
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AL = + ^^)-50ci72 = .25 cm 



Thus we find a net positioning error of only one quarter of a 
centimetei Now imagine that the fiducials are only spaced by 
10 cm (instead of 30 cm) thus increasing the extrapolation 
coefficient L/lz» from 167 to 5, and decreasing the number of 
accurately resolvable points between fiducials 1/e, from 375. to 
125 Under this more pessimistic scenario, we would find the 
net object plane positional error increased to: 



AL = ^ .[III + li^)- 50 cjn = 1.7 CT? 
.5 V 4 ^ lOcra^ 
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Qualitatively, we can attribute this unacceptable error result 
to a combination of the inci eased extrapolation coefficient due 
to closely spaced fiducials far from the point of interest, and 
a camera field of view poorly matched to the narrow field of 
fiducials. The lesson of this example, is to provide a rich 
field of fiducials that would largely fill any likely camera 
field of view, and not be widely separated (relative to their 
spacing) from the points of interest. 

For a field with a high density of fiducials, there can be 
assumed to be ah extent 1, approaching the total field of view 
of the video camera, given by 21pov. Let us employ the effective 
resolution R'.ot defined earlier, for an image plane error e, as: 
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Then we can write, assuming a high density of fiducials, the 
estimation error 16, 20 for the case of 4)=0. as: 



= (1 ^ -rY^)i (4>=0) 20 



Everything else being equal, it can be seen that the eiror will 
be inversely proportional to the camera field of view i^ov 
What's more, the required number of physical fiducials to 
achieve a given H within the camera's field of view, will 
decrease according to Ifov"'' 



Regardless of the placement of fiducials, however, a 
situation in which the distance between the camera and the 
object plane is allowed to dwindle, will necessarily decrease 
the physical spacing between observed fiducials., If this is 
allowed to occur without a corresponding decrease in L, then an 
increased error due to a large extrapolation coefficient will 
occur.. A reasonable strategy for avoiding this circumstance, is 
to place the tracking cameza as close as possible to, and 
parallel to the direction of, the eye viewing the computer 
generated display, so that the human will nevei be looking in 
a location too far from the field observed by the video camera. 
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Expected Pprformance of Proposed Configurations 

In Table I, three systems are summarizedr '^The system designated "A" 
corresponds to a camera based on the imputer^ (VLSI Vision Ltd.) which 
includes a low-resolution video camera. The lens provides a (horizontal) 
f ield-of -view of 35', requiring a density of fiducials of 200 per square 
meter in order to be reliable at distances from the camera to the board as 
small as 2» = 30cin- Case "B" corresponds to a reasonably high resolution 
camera with a 60* f ield-of- view, still using a board containing 200 fiducials 
per square meter.. Case "C" represents the implementation of the same camera 
with a wide angle lens (90*). but in which the density of fiducials is 
decreased from 200 to 50, possible with the wider lens f ield-of- view.. 



Tabled 

Specifications for Three Proposed Systems 





A 


B 


C 


Pixel Resolution (H x V) 


256 X 256 


512 X 480 


512 X 480 


Density of Fiducials p 
(ff per square meter) 


200 


200 


50 


Field of View (horizontal) , 0„v 


35' 


60' 


90* 


ifOV 


.315 


. 577 


1. 


1 pixel error in image plane, e 


. 00154 


. 00144 


. 0025 



In Table IJ. the performance of each of these systems is evaluated for 
six different caL4?fiU' . We look at three different camera- board distances z,, 
of 60 cm, 30 G]n^iind-2a cm. In each case, we have explored two different 
values of L, the distance between the point at which the camera's axis 
intersects the board, and the point of interest at which the error is 
evaluated. In each case one value of L is reasonably small, so that the 
point of interest is near or inside the field- of -view of the camera itself. 
The other corresponds to a substantial extrapolation, thus revealing the 
worst case performance. In each example, we have assumed that the axis of 
the camera is approximately normal to the board. 
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Table II 

Table of Er'x^r Performance for Cases 1 B, and C 
(All lengths, in centiweters) 







L i Case 


N 


p. 








1 


Extrap 




1 


60 


20 


A 


21.4 










., f 


( 90) 


(.05) 


2 


B 


72. 










70 


( 47 ) 


(.01) 


3 


C 


A 










1. 22 


(.27) 


(.02) 


1 ^ 


60 


100 


A 


O 1 A 
^ i *4 










.37 


4.5 


.7 


1 ^ 


B 


( C - 










.70 


2. 36 


. 11 


6 


C 













1,22 


1.37 


.09 


7 


30 


20 


A 


5.. 4 


.21 


.. 09 




... WWO 


. 0 J. / 




. 15 


8 


B 


18. 










. O 1 o 


(. 99) 


( . 03 ) 




C 


13 0 










1. 15 


( 58) 


(.03) 


- 

10 


30 


75 


A 


0 . I 


9 1 


094 


. 03 


005 


.317 


7 9 


2. 0 


11 




B 












.675 


3.7 


25 


12 


C 


1 % ^ 










1. 15 


2.2 


20 


1 

13 
14 

15 


20 


15 


I A 


2.. 4 


'7 O 

r o 


D / 


0 1 




23 


3. 3 


. 40 


B 


8.. 


.. 04 








6?5 


1. 20 


. 04 


C 


6 . 








. 003 


1, 03 


(.72) 


(.04) 


16 


20 


50 


A 


2.. 4 


. 78 


. 57 


.31 


. 09 


23 


11. 0 


3- 4 




B 


8. 


.. 04 


. 015 


. 003 




.625 


4.. 0 


.26 


18 


C 


6. 


,. 15 


. 06 


.. 017 


. 003 


1. 03 


2.4 


.22 















Table n that system '•A' obtains unacceptable errors in the case 

of substantial extrapolation (lines 10 and 16), especially at smaller values 

of 2,. 
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Abundance of Fiducials 
In Table 11. ti column labelled 'N" aho, the expected number oi 
fiducials that would be seen by the ca«r^ dependent on its field o£ vie» 
1 the density of fiducials on the board P, and the speo.f.c distance 
beween the camera and the board z.,. The value of N 

determination of the expected object plane positioning error AL due to the 
TZZli accuracy In parameter estimation usin. multiple -0^"^-* 
Inurements. H, the mean number of fiducials In the camera s freld of v.ew^ 
liso deter»ines the probability o£ less than four fiducials bern. present rn 
the field of view at an arbitrary time. 

in Table U. P. denotes the probability of a soft iiOucUl depletion 
faiiura, »hlch occurs when the tracking camera has less than four fiducials 
within Its field of view. With only three fiducials in the tracklM camera, 
there is the possibility of misconstruing the direction of an angular motion 
when crossing certain boundaries However brief failures of thrs sort will 
not usually lead to tracking errors. If an ambiguity In tracUng were to 
occur during this Period, It would be Immediately resolved followrng the 

anauisition of a fourth fiducial 

P denotes the probability of a .ar. /i.uc.ai de^Jetron /aii.re whrc^ 
occurs when the tracking camera has less than three fiducials wlthrn rt. 
field of view- «lth only two fiducials in the tracking camera, it r 
impossible for the tracking algorithm to operate properly because th. 

: n Of six unknowns is impossible from the four numbers 
two data points Nevertheless, an algorithm can be instructed to perfor 
tracking on the basis of an a priori model In which certain head motion. 

as looking back, and forth, are deemed more likely than others sue a, 
atooping down with the head looking upwards, or standing more * " 
talU Thus b«l*f period. In such a state may not cause an oblectionabL 
tracking failure. Also, during such a state, the observation of the posl ro 
twoTldUclals Will enable the tracker to. almost ^^^^ -^f^^ ^ 

of a third fiducial when It comes Into view (however the tracker will neve 
b, sure of proper correspondence until four fiducials are in 

P. denote, the probability of a serious iUuo.al ^fJ.-""" j"'"" 
Which :ccurs When the tracking camera has less than two fiducials wit in 
field of view. With only one fiducial in the tracking camera, it 
IlffLl for the tracking algorithm to operate with any degree of accurac 



since the solution of .«ix unknowns is utterly impossible from the two numbers 
characterizing one o^^a point. Nevertheless, L algorithm can perform 
extremely crude tracking on thie basis of a-verr' oversimplified model of a 
priori head motion Since one fiducial remains in the field of view, when 
additional fiduciais are observed, the identification of those points will 
be simplified greatly by the continuing identification of the one. point. 
However initial misident if ication of a second and third fiducial may occur 
if significant changes in head orientation occuz during the period of a 
serious fiducial depletion failure. 

Ps denotes the probability of a catastrophic fiducial depletion failure, 
which occurs when the tracking camera has absolutely no remaining fiduciais 
within its field of view, at which point tracking stops. When a fiducial 
then re-enters the camera's field of view, the tracker may be able to make 
an educated guess as to its identity, but will generally have no confidence 
in that guess until the point at which four fiduciais are again in view.. 

The calculation of these failure probabilities is based on a worst-case 
model in which the placement of fiduciais is totally random, with the 
specified average density, so that the number of observed fiduciais is 
governed by Phoisson statistics In a real fiducial pattern, clumping of 
fiduciais and areas of fiducial depletion will be avoided- Thus situations 
in which the expected number of fiduciais in view N, exceeds 10, have zero 
probability of even a soft fiducial depletion failure, and the corresponding 
cells in Table II are left blank For 6<N<10. the numbers for the failure 
rates are somewhat pessimistic for the same reason. For N<6. the probability 
or tiaacial depletion failures swells, making the task of reliable tracking 
cenuous (lines / , 10, 15, Iti) or impossible (lines IJ . 16).. 

It Should also be pointed out that the calculation of the expected 
number of fiduciais in view N, is based on the camera axis being normal to 
the object plane: ^^Z. For the same z^, an inclination of the camera's 
axis with respect to the board, will increase N, the expected number of 
fiduciais within the field of view. For instance, using camera C in line 
18 if the helmet were tilted by an angle of 30^ the expected number of 
fiduciais in the camera s view N. would be doubled to 12 

• Ln an/ case, nowever, a given camera field of view, and a given density 
ot fidu:ial5 will deTine :i minimum working distance 2„,m, below which 
riducisl deolrstion failures are likely to become troublesome.. For a camera 
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with a horizontal acceptance angle ©fov with a dep^ity of fiducials p, the 
expected number of fiducials In the camera's field view will exceed five, 
as long as a, exceeds 3m» given by: 

1.3 

' "sfp tan{e^^/2) 

In this equation, l/v'p can be viewed as the "average distance between 
fiducials." defined as the spacing between fiducials that would be obtained 
if they were arranged in a rectangular grid pattern- It can be seen that an 
increase in the camera field of view can dramatically decrease the number of 
fiducials required to be placed on the board. Attempting to accommodate 
small working distances, on the other hand, will rapidly drive up the number 
of requixed fiducials.. 
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EXHIBIT D 



To: nilzell@espresso.rt.cs,boeing.com 

cc: janin@espresso-rtxs.h ingxom 

Subj: Re: ARS Program — 

David: 

Thanks for the note. Norm Tarleton is a nice guy, and intelligent, but he 
powerless to help you. 

We are working very, very hard to complete a production tracker 
demonstration for you mid next month. We will be able to produce the 
tracker at a "reasonable" price. 

The VFE II is complete, tested, documented, and robust. 

A copy of a status report I sent Jim Bath Is appended below. 

Thanks for your help. 

Jim 



Date: Tue, Mar 26, 1996 5:00 PM EDT 
From: WolvertonJ 

Subj: Confirmation of Work Direction 

To: jbath@p04.mn 1 Z.honeywell.com 

cc: lewandowski..ron@mnl 5-gw.mavd.honeywell.com, 

bender_barb@mnl 5-gw.mavd.honeywelLcom 

Ron & Barb please forward to James Bath.. Your group has such an 
interesting email system that we may need to revert to snail mail and 
telephone tag to establish reliable communication. Enjoy the wonderful 
world of "Quick & Dirty Operating System" (DOS). 

James: 

Confirming our near term ARS tasks: 

1 . TriSen is completing integration of the "C" program on a 486DX 66 PC; 
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the program contains an assembly language module for contrpl ofthe 
parallel port, (which allegedt neets IEEE specification 12& 

The programming tasks include those for the Video Feature Extractor 
(VFE) Gate Array Logic chips; the chips provide the necessary flexibility 
for this stage of development. We have completed and tested the 
programming for VFE II. We have excellent documentation for that 
programming. 

We have written a draft Applications Program Interface (API) for 
operation with the Boeing software. Adam Janin and we believe it will 
work, but the proof will be demonstrated in Seattle. 

We have a well documented Forth language implementation of initial 
correspondance. We will transfer this implementation to "C" after 
completion of the higher priority programing. 

We have completed a significant testing regime and are completely 
rewriting the "C" implementation of the clumping, centroiding, lens 
aberration correction, and tracking modules. These are being documented 
as they are being written with the participation of our "C" programmer. 

(2) We, and you, are awaiting the delivery of the backpack 486 computer 
with its new parallel port. We are hoping the "C" program integrates 
easily in that computer processor, but are concerned about the new 
parallel port. 

We are also concerned about the "real" battery voltage available. We will 
need to convert it, using a power supply on the VFE, to drive the head 
mounted TV camera and the VFE electronics. Depending upon the value and 
"droop" of the voltage, this integration may be easy or require a complete 
redesign of that portion of the VFE. 

(3) We are also preparing to integrate with the Flex "wearable computer." 
The same risks apply. 

(4) We have designed, and have in fabrication, the production prototype 
wiring boards. They will be robust, stable, mechanically sound, and 
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provide both "white dot" and "black dot" instantiations. 

We will need to determine the lighting, conditions.at Boeing, user color 
preferences, etc. to choose final colors and lighting supplement (if 
required), 

(5) We have completed and checked out two ("white dot" and "black dot") 
instantiations of VFE IL The final schematics are being entered into the 
computer to generate "good copies" and net lists. 

Have a fun telecon with Boeing tomorrow. 



Jim 
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EXHIBIT E 



Communications between the VFE and a P,C. 

Following is a description of communication protocol applicable to 
the Video Feature Extractor, version 2, using a P.C, with an Enhanced 
Parallel Poit (EPP) interface. This description applies to the 
circuitry as modified according to changes specified on March 11 and 
March 14, 1996. Also note that the "clear-to-end-of-f ield" command has 
been eliminated as of the 7-9-96 version of U7. 

Control of the VFE to PC interface is performed by U7 which also 
controls output from the FIFO's {U4 and U5), and U6 to the parallel 
port, and which additionally directs input to the VFE from the parallel 
port to U6 and U3 . The FIFO^s (174 and U5) have all 8 data bits hooked 
up to the bi-dixectional port in the computer, as does US. U3 receives 
input only from data bits 1-5 of the parallel port, and the control 
chip, U7, is. hooked up to data bit 1 of the port, in addition to several 
control signals. 

There are 4 control signals from the computer feeding U7 , and U7 
sends two control signals to the computer (in addition to the hookup to 
data bit #!)• The EPP control signals driving U7 are: 

1) /Datastrobe or /DS4 from EPP pin 14 driving U7 pin 9. 

2) /Write or /W from EPP pin 1 driving U7 pin 10. 

3) /Addrstrobe or /DS3 from EPP pin 17 driving U7 pin 15 (as per 
the March 11, 1996 modifications). 

4) /RESET or ALT from EPP pin 16 driving U7 pin 8. 
The VFE sends to the computer three control signals: 

1) /WAIT from U7 pin 23 to EPP pin 11. 

2) ERROR from U7 pin 22 to EPP pin 15. 

3) /INTERRUPT from U6 pin 22 to EPP pin 10. 

Of these signals, /DS4, /DS3, /W, and /WAIT, are interlocked 
handshaking signals involved in EPP reads or writes at device addresses 
PORT+4 or PORT+3. /ALT is controlled by writing bit 3 (04h) of device 
address PORT+2- ERROR is read as bit 4 {08h} of device address PORT+1. 
/INTERRUPT causes a parallel port interrupt (usually IRQ7 , interrupt 
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vector OFh) if interrupts have been enabled by settin«f bit 5 (lOh) of 
device address P0RT42 . 

Internally to the VFE, data to or from the EPF are controlled by 
the following control signals generated by U7. Output from U6 to the 8 
data bits of the parallel port are controlled by the signals OCl and 0C2 
generated by U7. The interrupt mode register (internal to U6) is loaded 
by U7 clocking the signal INTERRUPT-SET. Outputs from the FIFO's, U5 
and U4, to the parallel port data lines are enabled by assertion of 
/READ-1 and /RBAD-2 respectively {the falling edge of these signals also 
clock out the data just read). Data from parallel port data lines 1-5 
are latched into the garbage limit register (internal to U3) by U7 
clocking the signal /GARBAGE-SET. Finally, all data is cleared from 
both FIFO's, and data acquisition by U3 is halted (until the beginning 
of the next video field) by U7 asserting /FIFO-RESET. 

The signals 0C2 and OCl determine the output state of U6 as 
follows: 

• If 002 = 0 and 001=0, U6 does not drive the parallel port data 
lines. 

# If 002=0 and 001=1, U6 drives the parallel port data lines with 
all zeros (OOh) . 

♦ If 002=1 and OCl=l, U6 drives the parallel port data lines with 
the current time, 

# If 002=1 and OC1=0, U6 drives the parallel port data lines 3-8 
with zeros, drives bit 2 with status information (with a 1 
indicating that the FIFO's are at least half-full), and does not 
drive bit 1, allowing it to be driven by U7. 

Pz'ogramming Procedures 

Control of the above functions by the CPU is through byte or word 
INPUT and OUTPUT instructions at device addresses PORT through PORT+4, 
where PORT is the base address of the enhanced parallel port adapter, 
typically 378h. Following are valid output commands: 



♦ Outputting a 4 to PORT+2 will set the. ALT bit detected by U7 pin 8. 
Output ting a 0 to PORT+2 w- j. clear the ALT bit. The i itua of the ALT 
bit will affect the meaning of other input and &u-tput- operations to the 
VFE! 

♦ Outputting a byte to PORT+4 will initiate a wiite cycle in which W and 
DS4 are asserted, U7 responds by clearing WAIT after which the wiite 
cycle ends. If ALT=0, U7 clocks the INTERRUPT-SET signal so that the 
interrupt register in U6 is loaded from the parallel port data lines. 
However if ALT=:1, then U7 performs a master reset by pulsing FIFO-RESET. 

• Outputting a byte to PORT+3 will initiate a write cycle in which W and 
DS3 are asserted. U7 responds by clearing WAIT after which the write 
cycle ends. If ALT=0, U7 clocks the GARBAGE-SET signal so that the 
gaibage register in U3 is loaded from parallel port data lines 1-5* 

Following are valid input operations: 

• 16 bit data is read fiom the FIFO's by setting ALT=0 and performing a 
word input fxoiu PORT+3 which initiates a double read cycle which occurs 
in the following manner* First the EPP adapter asserts DS4 in response 
to which U7 sends out READl which causes FIFO #1 to present data onto 
the parallel port data lines (this is the high byte of the data word). 
U7 clears WAIT in response to which the EPP adapter clears DS4. Then 
the EPP adapter asserts DS3. U7 then pulses READ2 causing FIFO #2 to 
drive the data lines with the low byte of the data word. U7 clears WAIT 
in response to which the EPP adapter ends the entire input cycle and 
sends the 16 bit data to the CPU. 

The format for this data word is described in a separate document. 
In the case that an input operation is initiated when there is no data 
present in the FIFO, then U7 , rather thjan enabling output from the 
FIFO's, instructs U6 to drive all 8 parallel port lines with zeros for 
each byte transfer.. Thus an input data word of 0 signifies an empty 
FIFO (there is no possible data word which will be exactly 0). 

While performing a word input from PORT+3 is the recommended form 
of inputting data from the FIFO's, byte inputs from PORT+3 or PORT+4, or 
word inputs from PORT+4, (with ALT=0) will cause some of the same 
strobes to clock data out of the FIFO's. For instance, a 16 bit input 
from PORT+4 will cause two inputs to occur via the DS4 strobe, which 
will confuse the VFE. The VFE expects an input cycle via DS4 to be 
followed by an input cycle via DS3. So a word input from PORT+4 will 

3 



set an error condition in the VFE, as will most, other combinations of 
byte inputs when AL'. J. 

♦ With ALT=:1, inputting a byte from POBT+3 initiates a read cycle in 
which DS3 is asserted. In this case the VFE status is read, in which U7 
drives data line 1 with status information and U6 is instructed (by 
0C2 = 1, OC1=0) to drive data line 2 with status information. Data lines 
3^8 ar e each driven zero by U6 . The resulting status code is 
interpreted as follows: 

0 No data present in FIFO. 

1 Data ready in FIFO. 

2 FIFO is full and has overflowed. If no data reads have occuzx'ed 
since the onset of the overflow, then up to 1024 valid data words 
may be read, if desired. Following that (or instead of that) a 
master reset is strongly recommended in order to prevent skewed 
data or video fields with missing runs. Note that if a FIFO 
overflow occurs followed by one or more FIFO read operations so 
that the FIFO is no longer full, then this condition will no longer 
be read. However the error condition (see below) will be set until 
a master reset is performed. 

3 FIFO is at least half full (512 data words present, for a VFE 
containing the CY7C425 chips in the U4 and U5 sockets). 

• Inputting a byte from PORT+4 in the case that ALT=1, causes a read 
cycle in which the EPF port asserts DS4. This causes the VFE time to be 
read, in which U6 is instructed (by 0C2=1, 0C1=1) to drive the 8 data 
lines with the 8 most significant bits of the VFE time base. 

• Inputting from PORT+1 is used for reading the ERROR bit from pin 22 of 
U7. Its value is presented as bit 4 (08h) of the input data. The 08h 
bit indicates an error condition. An erior condition signifies the 
detection of a FIFO data skew or a FIFO overflow. The error condition 
must be cleared by performing a master reset, as described above. 
However if the error is a result of a FIFO overflow, it is possible to 
first read out any valid data residing in the FIFO. 

These input and output operations are summarized in the following 
table. 



4 





PORT 


Byte or 


1 

' ALT 


: Opexation 


1 J)at» format 






Word 


1 bit 


• - 






POST+2 




X 


Set /reset ALT 


xxxx .xA::x 


WRITE 


PORT+4 


BYTE 


0 


Interrupt Set 


nil nil 


1 WRITE 


PORT+4 




1 


Master Beset 


XXX K XXXX 




POET+3 


BYTE 







' Garbage Set 


KKxd GGCJG 1 


t HEAD 


POjRT+.3 I 


5?0KD 1 




Head PIf 0 data 


(i^ee text} | 


PI? AH 


x^KTrii j 


BYTE 


1 


Read STATUS 


0000 ooss 1 


READ ' 


P0K'r'f4 


BYTB 


1 


Read TIME 


J- -L •)( A I .1 2 1 K 


BEAD 


POBT+l 






Bead EEBOa 

1' 11 '•- 


xxicx Exxx 1 



{x^" don't care") 



D 



EXHIBIT F 





n 



c 



m. := /W15*S1*/S2 ^ "?2*S1*S2 + Sl*/S2*5ll«ll2«li^lJ4*«5*SI6*Sfr ; 

/xHlTO : = CaS=S23 EJS=Sl*/52 + /Sl*/S2 +S1*S23 £C=CluE3 ; 
iiZ := /lifZ*Sl^/B2 + ¥3*S1*S2 + Sl*yS2*»l*Si2«Si9«i«««5*8B*S7 ; 
- /2fi2'93 := Z(S=S21 L^SVU/SZ^m. + /Sl«/S2 +S1*S2J CC=CUD i 

/3iS4Sl«/S2 + T4«S1*S2 Sl*/S2*»l*IB*S3*il4*H5*tlB*»7 ; 

/xW3V4 := C<^S21 tS=Sl*/S2*«l*^ + /Sl*/S2 +S1*S21 CCsC£S3 i 

/xH4?5 := COS=S23 rS=Sl*/S2*»l*S2'Wi3 + /Sl*/S2 -i-Sl«S23 CC=CLS3 i 
MS := /W5«l*/S2 + ^^1«S2 + Sl*/S2«Sl*S2*»3»W4*S6*8i8»W7 ; 

/sS5¥8 := C<ffi=S21 C2=S1»/S2»»1*IG*S3*?I4 + /51*/S2 +81*823 CC=G£23 ; 
!S := /W6^1«/S2 + W*S1»S2 f Sl*/52*»l*a2*S83»il4»fi5*aB*W7 J 

/XSSW := lQg=S23 £g=Sl*/52*Sl«»2*!l3«W4*Si5 + /51*/S2 -151*823 £<;=C1K3 
m := m*Sl*/SZ 4- V8*S1*S2 Sl*/S2*Hl*li2*HS*W4*fl5*»6«H7 ; ^ 

/xii?78 i:05=S23 i;S=Si»>S2«9il«»2*S3«tl44SiS«f» /Sl«/B2 tSl«S2J r&=aL 

xTl := /?i tC=/Hll 1 C(aS?Sl«8S3 ; 

TO := /72 [C!=/yi3 ; 

73 := /TO C&:/?23 » 

?4 := m tC=/V33 ; 

W5 .-= /?5 CC=/?43 ; 

JB := CC=/753 J 

77 ;2s CC=/?83 ; 

¥B /V8 tCs/Wl ; 

i^ST =: 7P0L8£ ; Jus-t In order -to r«sse^ -blue 7 coantex! \ 



GX :=r rOSsOFI'3 fCpCLKl ; Coafisiaratioa equation really toe 0VSBSI2S 

miSBISIZS := 1IHiaSBSZ2S*(64«/fi& + GS*/W * 4S2»/W3 Gl*/^ ) /SI 
/0V2SSIZE :s /07SBSlZB*(G8«/fr7 -fr GT*/1i8 + G»*Al5 + 65*/f?4 )*/7P0I.S» /SI 

/68«/S7«/66*/B5 ; «< -fco disahle cfaackiiitf vith t lapat 

CAPWAfig = OTQSIZK + IIHI3KBSI2S •i-/TO»/V7«/76«/V5*/V4 : 



TITL2: 013 - Output, tjntrol cMp for CPD iaterf - to WE 



CPH Output to register 2: 

vaJue (iea) Stit5be2 Strobel 

9 9 8 

1 0 1 

2 18 

3 11 

18 (-)- abo^e) 



effect 
idle 

Sets gEurbage llaits 

Beadout FIFO data 

Master Resett (Discard FIFO data) 

Enable P&rallel port interrupt 



CPO Bead froa register 1 DIFFBBEHT FOB ?KBSIOS 211 



Beg 1 value 
as read (hex) 

8 

IB 
28 
38 



48 (-^above) 
88 (-{-above) 



Status3 Statas2 Statusl Heasing 

idle 

BS^Y (After E Syms 
Data OTSBFLOfl 
FIFO is H&LF FOLL 

VFOLSE inactive 
0 TPDLSE active 

(also pulses low when HALF FOLL rises) 
(If enabled, bi-lo transition on this pin causes 
an interruixt tbroogh parallel port, IWW, when using LF 



9 


8 


8 


8 


8 


1 


0 


1 


0 


0 


1 


1 


1 







VEBSIOH * 4 DATS 18-27-95 



DBVICE GAL16V8 
IS 'Output' 
OPEBATOBS * + $ / 



PIHS 

GLK, OB-QID 1,11 ; 

/BSyHC. m, DB. /OWLW. /VPOLSE, /PHBOP 2.3,4.5,6,7 ; IHPOTS 

/STROBKl 8 ; In3E>ttt from par. port pia 14 

/STE0BK2 9 ; Input from par., port pin 1 

STATUS 1 15 ; Output to par. poart pin 13 

STATDS2 14 ; Oatput to par., port pin 12 

STAT0S3 18 ; Output to par. port pin 10 



/SKACODT 17 ; Clock ov^v^t of FIFO 
'/GPO-COBMAKD 18 ; ■ 

/MR 19 i . _ 

QXySSTWiBS 13 1 lataraal sto-te for debounclsg input si;robes 



.SQDATIOIiS 

IgOTSTBOBS := STBOBSl + STfiOBS:2 ; 



CPO-COMMAND := /aSAlX3^T*/CR^-COS^aAilI)*/Ma*GOTSTHOBE*/STEOBE2^^ 
-i- CFO-COHH&HBW^STROBB ; 

SBA35O0T := /BSAIX)irr*/CP0-C0mMn)*/MR*6OTSTB0BE*STROfflE2*/STS0BEl 
4- BHA00DT«G0TST50BS : 

m := /crU-C0!JMABI)*y!!8*G0TSTH0BE*STBOBjE2*STB0BEl + MR*GOTSTfiOK 
J/ + HR^/VFOLSS + PWROP : 

STATUS! := flE*/OVFL»t!STATOSl + Dfi*/OWLII*HSYHC 4 HF«/pVIL» ; 
STATl^2 := HF*/STATDS3 ■♦• HP*STAT0S2 + OTFLW ; 

/STATOSS := VPOLSS + BF*/STATlffi2 J 



C: \PLAQ\TRISEN\V¥E\CNTLATCH\DATAFK11T. TXT 



OOTPOT DATA ORGANIZATION - VERSION 2 (10-16-95) 



Order of 


Select lines 




Data 


organization - 


FIFO 


data 


lines 


Dl 


presentation 


S2 


SI 


D9 


D8 


D7 


D6 


D5 


D4 


D3 


D2 


1 


0 


1 


Pll 


P10 


P9 


P8 


P7 


P6 


P5 


P4 


P3 


2 


1 


0 


W7 


W6 


N5 


W4 


W3 


W2 


Wl 


P2 


PI 


3 


1 


1 


V8 


V7 


Y6 


V5 


V4 


73 


V2 


VI 


V0 


Parallel port 


data 


line 4» 


8 


7 


6 


5 


4 


3 


Z 


1 


* 


Parallel port 


DB-25 


pin It 


9 


8 


7 


6 


5 


4 


3 


2 


11 



* Read on status port, bit 80h 



P(ll-l) = Position of beginning of run 

W(7-l) = Width of run 

V(8-0) = Vertical line of run 



ue 



24 



si— o 
I 
I 

s2-<> 
I 



latcli o 



6HD 



2 

3 

4 

5 

6 

7 

8 

9 

10 

11 

12 



23 Q carry 



22 
21 
Z9 
19 
18 
17 
16 
15 
14 
13 ^ 



Vcc 



92 



P9 



P5 



P4 



_y \ 



elk 



HI BIT 3 -bit Counter & 2 hit latdi: 
-A /- 
\ / 




11' Ar^ 



TITLE: Bate Geaeration "^ntxol cMp for VIS 






1 


garbage^ 


2 




3 




4 


lay O 


6 




6 




7 




8 




9 




10 



\ / 



US 



20 
19 
18 
17 
16 
15 
14 



12 
11 



Vcc 



^etrac^ 

(sendit) V 4r f/fir^ 
o> ^if o-cUk V, y 



13 o-s2,^ 



>id *x} 
^8 ^0 «^ 
^ ^ 





15 



» ^ &v «». ->» 



L 



ill fe 



NT \ 



1 



C^ 3r 



-2 



C 



<4 



1 



c 



9> 
< 



1- -Y- 



3i 



0 0 



lis 



^ 00 ^ 



C 



J 



jg ^ C 



















i 


> <l 







n 
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C:\PIjAQNTBISSH\r^\CaTLiffCH\HlCHT3.PtQ 



TITLS: 07 - HI BIT 3-bii; Counter & 2 bit la-fcch for VFS horizontal oloch 

THIS IS the highest order chip, counts 3 hits but only latches 2 hits 
and implements other also, functions 
THIS IS FOR TBS ?BfiSION »2 CISGUI7 9-6-95 



VSBSIOH » 5 DATS 10-16-95 



DEVICS GM.16V8 
ES 'hicnt' 
OPEKATOBS » + 5 / 

PIHS 

CLK, 0E-GJ1I> 1.11 

/carry, /latch, /si, /s2 2.3,4.5 ; IHPOTS 

h8. hl0, hll. latched!. latched2. pie. pll. gate 12. 13. 14, 17, 18. 15, 16. 19 

SQUATIONS 

-hS := h9*/carr7 + /h8*car3cy ; 

iia := hia*/cara:y + /blS)*carry*h9 + hl0*/h9 ; 
hll := hll*/carrF + /hll*carry«h9«hl0 + hll*/hfl + hll* /hie ; 

latched 1 := latchedl*/latch + hie*latch : 
latched2 := latched2*/latch + hll* latch ; 

pl»=latchedl C0E=sl*/a23 : 
pll=latched2 rOE=sl*/s2I ; 

gate ^= «ate*/oarry + gate^/hS + sate«hll + h8«hll«carry ; 



C: \HJLQ\T2ISr ' TO\CSSTLATCH\CST8.. PLQ 



riTLS: 8 bit counter srith 8 bit latch tor 11 bit horisoatal couatax;, 7FS 06 

VEfiSION 3 fox the 8SH DATA COHFIGOBATIOH 

Version 2 corrects the polarity of tlie pi line (fifo ql) 

mSIQH * 3 DATE 18-11-96 



DEVICE 6AIi39718 
ES 'cnteiatch' 
OPKHATOBS * + $ / 

PINS 

CLK 13 

/SI. /S2, B9. /LATCE 2.3.4.5 ; I8F0TS 

/P1.P2./P3.P4.P5,P6.P7.P8.P9./CABRT 21.22.14.15.16.17.18.19.29,23 ; 
; /I 2 /I 234567 
; Wirad to fifo bits * 

Hl.fi2.fi3.E4.H5.H6,B7,H8 25.26,27,28.29,36,31.32 ; Buried counter bits 



EQUATIOSS 

HI := /HI CC=CLS:3 ; 

H2 /H2 CC=CLK3 CE=H13 ; 

B3 := /H3 CC=CI.K3 tS=Hl*H23 ; 

B4 := /H4 i:C=CI^3 [g=Hl*a2*H33 ; 

H5 := /H5 CC=CLK3 CK=H1*H2*13*H43 ; 

E6 := /H6 CC::CLS3 £E=H1«H2«H3*H4*H53 : 

Wl /m CC=CLK3 CK=H1*H2*H3«H4*H5*H63 ; 

HB := /B8 CC=CLX] £E=Hl«B2«H3*fi4«B5*16*H73 ; 

PI := HI CC=CLK3 CS=LATGH3 C0B=/S1*S23 ; 

P2 := H2 CG=CI,K3 CE=LATCH3 C0K=/S1*S23 ; 

P3 := H3 CC=CLK3 CI=LATCH3 C0B=S1*/S23 ; 

P4 := H4 CC=CLKI «=LATCH3 C0B=S1»/S23 ; 

P5 := H5 CC=CI,K3 CE=LATCH3 C0B=S1*/S23 ; 

F6 := H6 tC=CLK3 CE=LATCH3 tOK=Sl*/S23 ; 

P7 := H7 rC=CLK3 tl=LATCH3 E0E=S1*/S23 ; 

P8 := B8 £C=CLK3 ES=LATCH3 rOE=Sl*/S23 ; 

P9 := H9 CC=CLX3 tS=LATCB3 [0E=S1*/S23 ; 



CAREY :- /H1*H2*H3*H4*B5*BB*H7*B8 EC=CLK3 ; 



— C: \PLAQ\TBISr XVFgVSaTACTSLXDATACTBL.PLe 



/ITLS: 08 - Sa-ta Generation Contjrol chip £or 7FE 

Vesrsion 6 sends out the TBACX signal to do background tracking when 

/feature, and ri^i^t AFTSS the hsyms signal as long as feature 
is still on 

Version b for HSSf DATA CONFIGDHATIOS will drive the 78 signal 
onto the FIFOl line 

Sends out signals: SI, S2, and FIFO-CIX for inputting data 
into FIFO 



VEBSIOH » 6 DATE 10-13-95 



DEVICE 6AL16VB 

KS 'datacjtrl ' 
OFEBATOJBS » + « / 

PIHS 

CLS. 0S-G8D 1.11 ; Ground the A3S pin, if not using to disable outputs 
GASMGE. FSATUBS. /ES7NC 2.3.4 ; IHFOTS 

nSR 5 ; /ES S ; <«< Not used in this -version 

/VSYNC 7 ; veartlcal sync input 

9 ; input of odd/e-ven field from sync separator 
/FIFO! 16 ; output of 70 when S1*S2 

/SI, /S2, /FIFO-CLK, SESDIT. /LATCH, /BETHACE 12.13.14.15.17.18 ; 
/TRACK 19 ; To cause tracking of background irhile active 

SQOATIONS 

LATCH := FEA7DBE»/Sl«/S2«/RSTfiACS»/LATCfi 

^ Sl«/S2»/SSNDIT«/FIFO-CLK4c/P5AT0fiS«6ABBA^/LATCH ; « Comes on to 

; IMVALIDAT5 garbage data 

BETSACE := HSYNC*/FIF0-CLK*/S2#/SEHDIT + BETBACE»FEATOBE + BETBACE*7SYHC ; 
THACX = /FBATOHE + BETBACE*/HSYHC*/7SIMC ; 

51 := LATCa*/Sl*/BSTBACE + S1»/S2*/FI?0-CL»»/LATCH*/HSTEACE 

+ S2*FIF0- CLK*/S1 + S1«S2»/FIF0-CLK ; 

SESBIT := Sl'4'/S2«/FIFO-CLS*/FSATDBB*/6ABBA6B*/LATCH«/SBHDIT ; 

52 := Sl*FIF0-CLK*/52 + S2»/FIF0-CLS + S2*/Sl ; 
FIFO-CLK := Sl*/FIFO-CLK»SB»DIT* /LATCH + S24t/FIF0-CLX ; 

FIFO! = Va C0E=S1»S23 ; 



C: \FLAQ\TBIS^^\??B\PHaSBDlT\PHASSDET. PLP 

; . ■ i 



TITLE: Phase detector, version 2. for ¥FE 

Will someday become integrated «ritli tise horiz couBter/3atch for 
better utiliz&tlon. 

Bas separate GOHI and GOLO outputs, to Ise 
connected to C7 filter thru DIODES! 



TSBSIOM # 2 DATS 10-26-95 



DSVICS GAL 1678 
Sis 'PhaseBet' 
OPERATOBS * + * / 

PINS 

cue, OE-GHD 1.11 ; Ground the /OS pin, if not using to disable 

H18. Hll, /CSYHC. B9. /CARS! 3,2.4.5,6 ; CtOP^ 

GOHI, /GOLO, CSYMC2, 18.18.12.13 ; 

EQUATIONS 

CS7HC2 := CSTHC ; Latch in CSIHC 

CS7NC3 : = CS?iiC2 ; Delay CSYSC2 by one clock cycle 



GOHI := Hll»Hl»ic/cs?llC3«CSYaC2»/CABR7 -i- GOHI«H11«H10«/6OLO ; 
GOLO := Hll*Hia*H9*CABRY*/G0HI*(/CSINC2+CSYNC3) f GOL0*/CSYHC2 




^^tber Version 7 : flags all arusas as 6£BB&^ af-ber the vortical cooater 
has trapped around, and is less -than; 8 

DNI)ESSI2E is fiSSSTT ^hen H4.1i3.Si2.Hl = G4,G3,62.G1 
OfSESIZS ia SETT when S7.K6,?i5.»4 = 07.66.66.64 

Bew pizioat with version 3 
Hew pinottt with version 4 

VEBSIOii # 7 DATS Ift- 19-95 



KBVICS GAL39V18 
SS 'fflulti-cnt' 
OFEMTOBS * -f 9 / 

PISS 

CLK 13 

GS,G7.<36,€5.G4,G3,G2, /SI. /S2. Hll, /VPOLSE 8.7.8,9.18.11.1.2.3.4,5; IBPDTS 

; inpot on pin 14. usiiig OLIiC for state Ff : 
OVSBSIZS 34 ; state FF 
61 14 ; input 



^l,xWlV2,xSi2V3.xW3?4,3S4V5.x»5V6.x»6W,xW7Va 15.16.17.18.19.28.21,22; ranrSICi 
7vi./Wl,/te./W3./W4./W5./W8,/W7 35.36.37.38.39.40,41.42 ; OUSCS 0xjTM% 

72. 73. 74, 75. 76, 77, V8 25.26.27.28.29.30.31 ; BLMC'S - Vertical counter 
mOSBSIZS 32 ; ELHC 



outputs: 




91 



S2 



state 



action 



0 

1 
0 
1 



0 

0 

1 
1 



idle 

counting or send F 
sezui N 
send 7 



count N 
bold W 
load 7 



clear N0-H7 



fiQDATIOHS 



Wl := /W1*S1*/S2 + 72*S1*S2 + S1*/S2*H1*W2*W3»W43«6*H6*H7 ; 

/3d»172 := i:0S=S2] CK=S1«/S2 + /Sl«/S2 +S1*S23 IC^CUn ; 
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